Papers with probing task
Knowledge Enhanced Contextual Word Representations (D19-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
| Challenge: | Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge. |
| Approach: | They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism. |
| Outcome: | The proposed model improves perplexity, ability to recall facts and word sense disambiguation. |
VLStereoSet: A Study of Stereotypical Bias in Pre-trained Vision-Language Models (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing studies on pre-trained vision-language models have focused on measuring biases and stereotypes in a single modality. |
| Approach: | They extend a recently released stereotypical bias dataset into a vision-language probing dataset called VLStereoSet to measure stereotypical biased vision-linguistic models. |
| Outcome: | The proposed probing task measures stereotypical bias in vision-language models and its intra-modal and inter-modal biases. |
Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing probing datasets focus on knowledge about relations between words and entities. |
| Approach: | They propose a new probing task to evaluate word understanding directly using dictionary definitions of words. |
| Outcome: | The proposed task evaluates word understanding directly using dictionary definitions of words. |
Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Neural relation extraction models capture linguistic and semantic properties of the input, a recent study shows. |
| Approach: | They introduce 14 probing tasks targeting linguistic properties relevant to RE . they add contextualized word representations to enhance probing performance . |
| Outcome: | The proposed models achieve state-of-the-art on two datasets, TACRED and SemEval 2010 Task 8 . they show that the models capture linguistic and semantic properties relevant to the downstream task . |
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions. |
| Approach: | They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
| Outcome: | The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
M3: Multi-level dataset for Multi-document summarisation of Medical studies (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing summarisation systems are not up to such complex tasks, yet limited tools exist to determine where and why they are failing. |
| Approach: | They propose to use a dataset to evaluate the quality of summarisation systems in the biomedical domain. |
| Outcome: | The proposed model can be used to evaluate the quality of summarisation systems in the biomedical domain. |
Where Does Linguistic Information Emerge in Neural Language Models? Measuring Gains and Contributions across Layers (2022.coling-1)
Copied to clipboard
| Challenge: | Probing studies have explored where in neural language models linguistic information is located . standard approach is to focus on the layers whose representations give the highest performance on probing tasks . |
| Approach: | They propose a method that asks where task-relevant information emerges in the model by focusing on the layers that give the highest performance. |
| Outcome: | The proposed method confirms the expected ordering only for one of the pairs, indicating that the features that contribute the most to probing tasks are not as high-level as global metrics suggest. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
The Chinese Causative-Passive Homonymy Disambiguation: an adversarial Dataset for NLI and a Probing Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research questions whether these models really understand the meaning of natural language. |
| Approach: | They propose to transform the disambiguation of causative-passive homonymy (CPH) to a challenging natural language inference task using a pretrained transformer model RoBERTa. |
| Outcome: | The pretrained model RoBERTa performs poorly on the CANLI dataset . the model's internal representation of CPH is not captured in the model . |
Evaluating LLMs’ Capability to Identify Lexical Semantic Equivalence: Probing with the Word-in-Context Task (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the capability of large language models to identify lexical semantic equivalence are not currently being used. |
| Approach: | They propose to use the Word-in-Context (WiC) task to determine whether the meanings of a target word remain identical across different contexts to evaluate their capability. |
| Outcome: | The proposed method outperforms other LLMs in the Word-in-Context (WiC) task. |
Can Pre-trained Language Models Interpret Similes as Smart as Human? (2022.acl-long)
Copied to clipboard
| Challenge: | Simile interpretation is a crucial task in natural language processing. |
| Approach: | They propose a task to let PLMs infer the shared properties of similes by probing textual corpora and human-designed questions. |
| Outcome: | The proposed task outperforms pre-trained language models on simile interpretation tasks while still underperforming humans. |
Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models possess certain commonsense and factual knowledge. |
| Approach: | They propose to use pre-trained language models to predict masked words . they introduce a probing task with 13.6k m-word-prediction probes . |
| Outcome: | The proposed model performs poorly on the diagnostic dataset prior to any fine-tuning and fine-testing with distant supervision. |
Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to follow multi-constraint instructions in real-world applications. |
| Approach: | They propose to quantify the difficulty distribution of constraints by a novel Difficulty Distribution Index (CDDI) they find that LLMs are more performant when presented with constraints in a “hard-to-easy” order. |
| Outcome: | The proposed model is more performant when presented with constraints in a “hard-to-easy” order, compared with existing models with different architectures and sizes of parameters. |
LLMs Can Compensate for Deficiencies in Visual Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a strong language backbone in vision-language models compensates for weak visual features by contextualizing or enriching them. |
| Approach: | They investigate whether strong language backbone compensates for weak visual features . they use CLIP-based vision encoders to perform controlled self-attention ablations . |
| Outcome: | The proposed model compensates for weak visual features by contextualizing or enriching them. |
Reassessing Semantic Knowledge Encoded in Large Language Models through the Word-in-Context Task (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have propelled significant progress, extending their application across various domains including dialogue systems, text generation, translation systems, and beyond. |
| Approach: | They propose to use the Word-in-Context (WiC) task to reassess the semantic knowledge encoded in large language models (LLMs) they prompt LLMs to generate natural language descriptions that contrast the meanings of the target word in two contextual sentences given in the WiC dataset. |
| Outcome: | The proposed model significantly improves the classification accuracy of the two models. |
Leveraging In-Context Learning for Political Bias Testing of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing probing methods for evaluating LLMs with political questions have limited stability and are unreliable. |
| Approach: | They propose to use human survey data as in-context examples to query LLMs with political questions to evaluate their potential biases. |
| Outcome: | The proposed task improves the stability of question-based bias evaluation and may be used to compare instruction-tuned models to their base versions. |